Electronic apparatus for identifying content based on an object included in the content and control method thereof

ABSTRACT

An electronic apparatus includes a communicator circuitry, and a processor for obtaining multimedia data from an external apparatus via the communicator, identifying an object in at least one frame from among a plurality of frames included in the multimedia data, and identify a content corresponding to the identified object based on content guide information provided from a first server.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2018-0113658, filed on Sep. 21,2018, in the Korean Intellectual Property Office, the disclosure ofwhich is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an electronic apparatus and a control methodthereof. More particularly, the disclosure relates to an electronicapparatus identifying a content included in a multimedia data, and anelectronic system.

Further, the disclosure relates to an artificial intelligence (AI)system simulating functions such as cognition, determination, and thelike, of a human brain using a machine learning algorithm, and anapplication thereof.

2. Description of Related Art

In recent years, an artificial intelligence (AI) system realizingintelligence of a human level is used in various fields. Unlike arule-based smart system, an artificial intelligence (AI) system is asystem in which a machine learns and identifies on its own. In the AIsystem, a recognition rate is improved and user preferences are moreaccurately understood the more it is used. Thus, the rule-based smartsystem has been gradually replaced with a deep learning-based artificialintelligence system.

Artificial intelligence technology includes machine learning (forexample, deep learning), and element technology utilizing machinelearning.

Machine learning is an algorithm technology that classifies and learnsfeatures of input data on its own. Element technology is technology thatsimulates functions such as recognition, determination, etc., of a humanbrain by utilizing a machine learning algorithm such as deep learning,and the like, which may include technical fields such as linguisticunderstanding, visual understanding, inference/prediction, knowledgeexpression, motion control, and the like.

Various fields to which artificial intelligence technology is applicableare shown below. Linguistic understanding is a technology of recognizinglanguages and characters of human, and applying and processing therecognized human languages and characters, which may include naturallanguage processing, machine translation, dialogue system, question andanswer, voice recognition and synthesis, etc. Visual understanding is atechnique to recognize an object as if the object were viewed from ahuman sight, which may include object recognition, object tracking,image search, human recognition, scene understanding, spaceunderstanding, image improvement, and the like. Inference and predictionis a technique of identifying information to perform logical inferenceand prediction, which may include knowledge/probability-based inference,optimization prediction, preference-based planning, recommendation, etc.Knowledge expression is a technique of performing automatic processingof human experience information as knowledge data, which may includeknowledge construction (data generation/classification), knowledgemanagement (data utilization), etc. Motion control is a technique ofcontrolling autonomous driving of a vehicle and a robot motion, whichmay include a motion control (navigation, collision and driving),manipulation control (behavior control), etc.

For automatic contents recognition for TV content recognition, it may benecessary to construct program identification information and a title asa database. For example, for content recognition based on fingerprint,it may be necessary to acquire in advance identification information(fingerprint) of a content to be recognized, and map the acquiredidentification information with a title of the content and store it in aserver. Further, when identification information for a current screen isacquired in a TV and transmitted to the server, the server may identifywhether matching identification information is present, and provide aprogram title and related information of the matching identificationinformation to the TV.

However, for the database construction described above, an additionaloperation or apparatus for collecting program identificationinformation, connecting the collected program identification informationto the program title, and sending it to a database server may benecessary, from which a significant cost is incurred. Further, a cost isalso incurred when the TV transmits a fingerprint, etc., to the serverand acquires a program title, etc.

Accordingly, a method for minimizing the server use is demanded.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure address at least the above-mentioned problemsand/or disadvantages and to provide at least the advantages describedbelow.

In accordance with an aspect of the disclosure, there is provided anelectronic apparatus including a communicator including a circuitry, anda processor configured to obtain multimedia data from an externalapparatus via the communicator; identify an object in at least one framefrom among a plurality of frames included in the multimedia data; andidentify a content corresponding to the identified object based oncontent guide information provided from a first server.

The processor may be further configured to: obtain a digital videofingerprint based on the at least one frame; control the communicator totransmit the obtained digital video fingerprint to a second server; andobtain content information corresponding to the digital videofingerprint from the second server via the communicator.

The processor may be further configured to: identify a type of theexternal apparatus; and based on the type of the external apparatusbeing a predetermined type, obtain the digital video fingerprint basedon the at least one frame.

The processor may be further configured to: control the communicator totransmit either one or both of the identified object and the identifiedcontent to a third server; and obtain an advertisement corresponding toeither one or both of the identified object and the identified contentfrom the third server via the communicator.

The electronic apparatus may include a storage, wherein the processormay be further configured to: identify the object in the at least oneframe based on an object recognition model stored in the storage,wherein the object recognition model may be obtained by training aplurality of sample images and a plurality of objects included in theplurality of sample images through an artificial intelligence algorithm.

The processor may be further configured to retrain the objectrecognition model based on information relating to the object and thecontent.

The processor may be further configured to: apply an optical characterreader (OCR) to the at least one frame, from among the plurality offrames, and identify a text; and identify the content based on theidentified text.

The processor may be further configured to: identify a type of theexternal apparatus; and based on the type of the external apparatusbeing a predetermined type, identify the object in the at least oneframe.

The electronic apparatus may include a display, wherein the processormay be further configured to: control the display to sequentiallydisplay the plurality of frames; and identify the object in a displayedframe from among the plurality of frames.

The object may include any one or any combination of a title of acontent corresponding to the at least one frame, a reproduction time ofthe content, channel information of the content, and a characterincluded in the at least one frame.

In accordance with another aspect of the disclosure, there is provided acontrol method of an electronic apparatus, the control method including:obtaining multimedia data from an external apparatus; identifying anobject in at least one frame from among a plurality of frames includedin the multimedia data; and identifying a content corresponding to theidentified object based on content guide information provided from afirst server.

The method may further include obtaining a digital video fingerprintbased on the at least one frame; transmitting the obtained digital videofingerprint to a second server; and obtaining content informationcorresponding to the digital video fingerprint from the second server.

The obtaining the digital video fingerprint may include: identifying atype of the external apparatus; and based on the type of the externalapparatus being a predetermined type, obtaining the digital videofingerprint based on the at least one frame.

The method may further include transmitting either one or both of theidentified object or the identified content to a third server; andobtaining an advertisement corresponding to either one or both of theidentified object or the identified content from the third server.

The identifying the object may include: identifying the object in the atleast one frame based on an object recognition model, and wherein theobject recognition model may be obtained by training a plurality ofsample images and a plurality of objects included in the plurality ofsample images through an artificial intelligence algorithm.

The method may further include retraining the object recognition modelbased on information relating to the object and the content.

The identifying the object may include: applying an optical characterreader (OCR) to the at least one frame from among the plurality offrames, and identifying a text, and wherein the identifying the contentmay include identifying the content based on the identified text.

The method may further include identifying a type of the externalapparatus, wherein the identifying the object may include, based on thetype of the external apparatus being a predetermined type, identifyingthe object in the at least one frame.

The method may further include sequentially displaying the plurality offrames, wherein the identifying the object may include identifying theobject in a displayed frame, from among the plurality of frames.

The object may include any one or any combination of a title of acontent corresponding to the at least one frame, a reproduction time ofthe content, channel information of the content, and a characterincluded in the at least one frame.

In accordance with another aspect of the disclosure, there is providedcontrol method of an electronic apparatus including a model learningpart, the control method including: obtaining multimedia data from anexternal apparatus; based on an object recognition model obtained bytraining a plurality of sample images and a plurality of objectsincluded in the plurality of sample images through an artificialintelligence algorithm, identifying whether an object is recognized inat least one frame from among a plurality of frames included in themultimedia data; based on the object being recognized, identifyingwhether a content corresponding to the recognized object is recognized;and based on the content not being recognized, obtaining a digital videofingerprint based on the at least one frame; transmitting the obtaineddigital video fingerprint to a server; and obtaining content informationcorresponding to the digital video fingerprint from the server.

The obtaining the digital video fingerprint may include: based on thecontent not being recognized, identifying a type of the externalapparatus; and based on the type of the external apparatus being apredetermined type, obtaining the digital video fingerprint based on theat least one frame.

The predetermined type may be any one or any combination of a set-topbox, an external content server, and a broadcasting server.

The identifying the type of the external apparatus may be performedbefore transmitting the obtained digital video fingerprint to theserver.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses one or more embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, and advantages of embodiments will be moreapparent from the following description taken in conjunction with theaccompanying drawings, in which:

FIG. 1A is a block diagram illustrating an example configuration of anelectronic apparatus, according to an embodiment;

FIG. 1B is a block diagram illustrating in detail an exampleconfiguration of an electronic apparatus, according to an embodiment;

FIG. 1C is a block diagram illustrating another example configuration ofan electronic apparatus, according to an embodiment;

FIG. 2 is a diagram briefly illustrating an entire system, according toan embodiment;

FIG. 3A is a diagram illustrating an operation according to objectidentification, according to an embodiment;

FIG. 3B is a diagram illustrating an operation according to objectidentification, according to an embodiment;

FIG. 3C is a diagram illustrating an operation according to objectidentification, according to an embodiment;

FIG. 4 is a flowchart illustrating a process of a content recognitionmethod, according to an embodiment;

FIG. 5 is a block diagram illustrating a configuration of anotherelectronic apparatus, according to an embodiment;

FIG. 6 is a block diagram illustrating a learning part, according to anembodiment;

FIG. 7 is a block diagram illustrating a response part, according to anembodiment;

FIG. 8 is a diagram illustrating an example in which an electronicapparatus and an external server are interlocked with each other tolearn and identify data, according to an embodiment; and

FIG. 9 is a flowchart illustrating a control method of an electronicapparatus, according to an embodiment.

The same reference numerals are used to represent the same elementsthroughout the drawings.

DETAILED DESCRIPTION

The embodiments of the present disclosure may be diversely modified.Embodiments are illustrated in the drawings and are described in detailin the detailed description. However, it is to be understood that thepresent disclosure is not limited to a specific embodiment, but includesall modifications, equivalents, and substitutions without departing fromthe scope and spirit of the present disclosure. Also, well-knownfunctions or constructions may not be described in detail if they wouldobscure the disclosure with unnecessary detail.

Hereinafter, one or more embodiments will be described in detail withreference to the accompanying drawings.

An aspect of the disclosure is to provide an electronic apparatusminimizing the use of an external server to identify a content, and acontrol method thereof.

In accordance with one or more embodiments, an electronic apparatus mayidentify a content based on an object included in at least one framefrom among a plurality of frames included in a multimedia data, therebyminimizing the use of an external server in content recognition.

FIG. 1A is a block diagram illustrating an example configuration of anelectronic apparatus 100;

Referring to FIG. 1A, the electronic apparatus 100 may include acommunicator circuitry 110, e.g., communicator, and a processor 120.

The electronic apparatus 100 may perform communication with an externalapparatus. For example, the electronic apparatus 100 may include adisplay such as a TV, a desktop PC, a notebook PC, a smartphone, atablet PC, smart glasses, a smart watch and the like, and reproduce amultimedia data acquired from an external apparatus such as a set-topbox (STB).

Alternatively, the electronic apparatus 100 may be an apparatus withouta display, such as a speaker, a computer body, or the like. In thiscase, the electronic apparatus 100 may transmit a multimedia dataacquired from an external apparatus such as a set-top box (STB) to anapparatus provided with a display.

Alternatively, the electronic apparatus 100 may be an element separatedfrom a display apparatus, which may be an apparatus for performing mainfunctions, such as an image processing function, from among functions ofthe display apparatus, and transmitting only the image processing resultto the display apparatus. In this case, the display apparatus can beminiaturized, and the electronic apparatus 100 may acquire multimediadata from a set-top box, and transmit the acquired multimedia data tothe display apparatus. Alternatively, the electronic apparatus 100 mayacquire content from an external content server, a broadcasting stationserver, etc., process the image, and then transmit multimedia datacorresponding to the content to the display apparatus.

According to an embodiment, the electronic apparatus 100 may be anyapparatus that acquires a multimedia data from an external apparatus. Inan embodiment, the external apparatus may be not only a set-top box, butalso an external content server, a broadcasting station server, asatellite cable server, and the like. Further, the external apparatusmay be a universal serial bus (USB), a compact disc (CD), a gamemachine, a set top, etc., and may be any apparatus that can provide amultimedia data to the electronic apparatus 100. Further, the multimediadata may include at least one from among a video data or an audio data.

The electronic apparatus 100 may be an electronic apparatus identifyinga content from the multimedia content. The specific operations thereofwill be described in detail below.

The communicator comprising the circuitry 110 is an element forperforming communication with various apparatuses. For example, thecommunicator comprising the circuitry 110 may support variouscommunication schemes such as Bluetooth (BT), Bluetooth low energy(BLE), wireless fidelity (Wi-Fi), Zigbee, and the like. However,embodiments are not limited thereto, and the communicator comprising thecircuitry 110 may support any communication scheme in which wirelesscommunication is possible from among communication standards.

Further, the communicator comprising the circuitry 110 may include acommunication interface capable of wired communication with variousapparatuses. For example, the communicator comprising the circuitry 110may include a communication interface such as high-definition multimediainterface (HDMI), mobile high-definition link (MHL), universal serialbus (USB), red-green-blue (RGB), D-subminiature (D-sub), digital visualinterface (DVI), and the like, and may perform communication withvarious servers.

The electronic apparatus 100 may be connected with an externalapparatus, and acquire a multimedia data. In this case, the electronicapparatus 100 may be connected to the external apparatus through acommunication interface such as high-definition multimedia interface(HDMI), DisplayPort (DP), thunderbolt, universal serial bus (USB),red-green-blue (RGB), D-subminiature (D-sub), digital visual interface(DVI) and the like, and the communicator comprising the circuitry 110may include all wired communication interfaces. Further, the wiredcommunication interface may include not only a standard performing avideo input and an audio input through one port, but also a standardperforming a video input and an audio input through two ports.

However, embodiments are not limited thereto, and the wiredcommunication interface may be any standard capable of performing eitherone or both of a video input and an audio input.

The communicator comprising the circuitry 110 may include an interfaceof all communication schemes capable of performing wired communicationwith various apparatuses, as well as the wired communication interfacesdescribed above.

The processor 120 includes various processing circuitry and controls ageneral operation of the electronic apparatus 100.

According to an embodiment, the processor 120 may be implemented as adigital signal processor (DSP), a microprocessor and a time controller(TCON), but is not limited thereto. The processor 120 may include atleast one from among various processing circuitry such as, for example,and without limitation, a central processing unit (CPU), a microcontroller unit (MCU), a micro processing unit (MPU), a controller, anapplication processor (AP), a communication processor (CP) or an ARMprocessor, or may be defined as the corresponding term. In addition, theprocessor 120 may be implemented as a system on chip (SoC) with abuilt-in processing algorithm, and a large scale integration (LSI) andmay be implemented as a field programmable gate array (FPGA).

The processor 120 may acquire a multimedia data from an externalapparatus through the communicator comprising the circuitry 110.According to an embodiment, the external apparatus may be an apparatusthat provides a multimedia data corresponding to a content to theelectronic apparatus 100, such as a set-top box. That is, the externalapparatus may perform image processing of the content and provide amultimedia data to the electronic apparatus 100, and the electronicapparatus 100 may provide the acquired multimedia content to the displayapparatus without additional image processing or may directly displaythe acquired multimedia content.

Meanwhile, the external apparatus may include content information in themultimedia content and provide the multimedia data in which the contentinformation is included to the electronic apparatus 100. For example,the external apparatus may provide, to the electronic apparatus 100, amultimedia content in which a content title, channel information, etc.,are added to a frame corresponding to a current time point from among acontent of a total of 10000 frames. Further, the external apparatus mayinclude content information in a plurality of consecutive frames andprovide the plurality of consecutive frames in which the contentinformation is included to the electronic apparatus 100.

The processor 120 may identify an object in at least one frame fromamong a plurality of frames included in the multimedia data. Forexample, the processor 120 may identify a monkey in at least one framefrom among a plurality of frames included in the multimedia data.Alternatively, the processor 120 may identify a graphic data added bythe external apparatus in at least one frame included in the multimediadata, and identify a content title, channel information, etc., from thegraphic data. In an embodiment, the object may include at least one fromamong a character, a logo, a line, goods or text.

Alternatively, the processor 120 may identify an object from an audiodata included in the multimedia data. In this case, the processor 120may identify the object by a preset time period instead of identifyingthe object at a specific point such as a frame. For example, theprocessor 120 may identify a character based on a speech during a firsttime period of the audio data included in the multimedia data.Hereinafter, for convenience of explanation, it is assumed that anobject is identified in a frame.

The processor 120 may identify a content corresponding to the identifiedobject based on content guide information provided from a first server.For example, when a monkey is identified, the processor 120 may identifya monkey documentary corresponding to a monkey from the content guideinformation. According to an embodiment, the content guide informationmay include at least one of a reproduction time, title, channelinformation, summary information, characters, story, of a contentprovided by a broadcasting station, etc., or an advertisement for thecontent itself, such as an electronic program guide (EPG). Further, thecontent information may include at least one of an advertisementimmediately before a reproduction time of the content, an advertisementduring reproduction, or an advertisement immediately after reproduction.

The processor 120 may acquire content guide information from the firstserver, and identify a content corresponding to the identified objectbased on the acquired content guide information. Alternatively, theprocessor 120 may search content guide information stored in the firstserver and identify a content corresponding to the identified object.

The processor 120 may, when identifying a content corresponding to theidentified object based on content guide information provided from thefirst server, further consider current time information and category.For example, when a monkey is identified, the processor 120 may identifya content corresponding to the monkey from among contents correspondingto a current time from the content guide information. Alternatively, theprocessor 120 may, when a monkey is identified, identify a contentrelated to monkeys from a documentary from among movies, documentaries,entertainment and dramas included in the content guide information. If acontent corresponding to the object is not identified in a current timeor in a specific category, the processor 120 may identify a contentcorresponding to the object in another time or category.

The first server described above may be an apparatus separate from theexternal apparatus. For example, the first server may be an apparatusproviding a content to the external apparatus. Further, the first servermay be an apparatus that provides not only a content but also contentguide information to the external apparatus. However, the electronicapparatus 100 may not acquire content guide information from theexternal apparatus. This is because an external apparatus such as aset-top box must be provided with an additional function in order toprovide content guide information to the external apparatus 100. Thatis, the external apparatus is an apparatus which focuses on a functionof providing multimedia data to the electronic apparatus 100, but thefirst server may be a general purpose apparatus capable of providingvarious functions as well as providing a content and content guideinformation. Accordingly, the processor 120 may acquire the contentguide information from the first server rather than the externalapparatus.

Meanwhile, the processor 120 may, when no content is identified, acquirea fingerprint, e.g., a digital video fingerprint, based on at least oneframe, control the communicator comprising the circuitry 110 to transmitthe acquired fingerprint to a second server, and acquire contentinformation corresponding to the fingerprint from the second server viathe communicator comprising the circuitry 110.

The processor 120 may not identify a content through object recognition.For example, the processor 120 may not identify a content correspondingto the identified monkey from the content guide information. In thiscase, the processor 120 may acquire content information through thefirst server using a fingerprint.

According to an embodiment, the second server is a server which stores aplurality of contents and a fingerprint for each frame included in eachof the plurality of contents, and when a fingerprint is acquired fromthe electronic apparatus, transmits content information corresponding tothe fingerprint to the electronic apparatus 100, which may be anapparatus separate from the first server. However, embodiments are notlimited thereto, and the first server and the second server may be ofthe same size.

In contrast, when a content is identified, the processor 120 may notacquire a fingerprint. This is because, when a content is identified byobject recognition, it is not necessary to recognize the content using afingerprint.

Further, the fingerprint is identification information capable ofdistinguishing one frame from another, which refers to intrinsic data ofeach frame. For example, the fingerprint is feature data acquired from avideo, image or audio signal included in a frame, and reflectscharacteristics inherent to a signal itself, unlike text-based metadata.Accordingly, the fingerprint is also referred to as fingerprint data,DNA data or gene data. For example, in a case of an image or a videosignal, the fingerprint may be a data expressing features such as amotion vector, color and the like.

The fingerprint may be acquired by various algorithms. For example, anaudio signal may be divided at predetermined time periods, and amagnitude of a signal of frequencies included in the respective timeperiods may be calculated. Further, a fingerprint data may be generatedby obtaining a frequency slope by calculating a difference of magnitudesbetween signals of adjacent frequency intervals, and quantizing thecalculated frequency slope by 1 when the slope is positive and by 0 whenthe slope is negative. However, embodiments are not limited thereto, andthe fingerprint may be acquired in various ways.

Alternatively, the processor 120 may, when no content is identified,identify a type of external apparatus, and when the type of externalapparatus is a predetermined type, acquire a fingerprint based on atleast one frame. Further, the processor 120 may control the communicatorcomprising the circuitry 110 to transmit the acquired fingerprint to thesecond server, and acquire content information corresponding to thefingerprint from the second server via the communicator comprising thecircuitry 110. That is, the processor 120 may identify the type ofexternal apparatus before transmitting the fingerprint to the secondserver.

For example, the processor 120 may, when no content is identified,identify the type of external apparatus, and when the type of externalapparatus is a set-top box, an external content server, a broadcastingstation server, etc., acquire a fingerprint and acquire contentinformation via the second server.

In contrast, when the type of external apparatus is a smartphone, a gameconsole, a digital video recorder (DVR), a DVD player, etc., theprocessor 120 may not acquire a fingerprint. For example, when theelectronic apparatus 100 is mirrored with a smartphone, the processor120 may acquire a screen displayed on the smartphone as a multimediacontent. However, a fingerprint corresponding to the screen displayed onthe smartphone may not be stored in the second server. Thus, even whenthe processor 120 transmits the fingerprint to the second server, theprocessor 120 may not acquire content information. Accordingly, when thetype of external apparatus is not a predetermined type, the processor120 may not acquire a fingerprint. In this case, the processor 120 maystop an operation for identifying the content.

In the above-described embodiment, a type of external apparatus isidentified after an object is identified, but embodiments are notlimited thereto. For example, the processor 120 may first identify atype of external apparatus, and when the type of the external apparatusis a predetermined type, identify an object in at least one frame.

For example, the processor 120 may identify the type of the externalapparatus, and when the type of the external apparatus is a set-top box,an external content server, a broadcasting server and the like, identifyan object. When the type of the external apparatus is a smartphone, agame console, a DVR, and the like, the processor 120 may not identify anobject. For example, when the electronic apparatus 100 is mirrored witha smartphone, the processor 120 may acquire a screen displayed on thesmartphone as a multimedia content. However, an object identified from ascreen displayed on a smartphone may not be related to content guideinformation. Further, a fingerprint corresponding to a screen displayedon a smartphone may not be stored in the second server. Accordingly,when the type of external apparatus is not a predetermined type, theprocessor 120 may not perform an object identification operation and afingerprint acquisition operation. In this case, the processor 120 maystop an operation for identifying the content.

In the above-described embodiment, a type of external apparatus isidentified, but embodiments are not limited thereto. For example, theprocessor 120 may identify a method for communicating with an externalapparatus. When the communication method is mirroring, USB, and thelike, the processor 120 may not identify an object and may identify anobject when the communication method is a remaining communicationmethod. Alternatively, the processor 120 may first identify an object,and when no content is identified, identify a method for communicatingwith an external apparatus. Further, when the communication method withthe external apparatus is mirroring, USB, and the like, the processor120 may not acquire a fingerprint, and may acquire a fingerprint whenthe communication method is a remaining communication method.Alternatively, the processor 120 may identify whether an object isidentified based on a multimedia data acquired from an externalapparatus.

The processor 120 may control the communicator comprising the circuitry110 to transmit at least one of an identified object or content to athird server, and acquire an advertisement corresponding to at least oneof the identified object or content from the third server via thecommunicator comprising the circuitry 110. Through this operation, thethird server may acquire a viewing history of the user, and provide anadvertisement based on the viewing history of the user. For example,when a large number of viewing histories of movies are present in theuser viewing history, the third server may provide a movie trailer, newmovie information, etc., to the electronic apparatus 100.

Further, the processor 120 may identify a content based on a viewinghistory of a user. For example, when a viewing history of movie contentsof a user of the electronic apparatus 100 is larger than a viewinghistory of dramas, the processor 120 may store user informationregarding the viewing history in a storage. Thereafter, the processor120 may identify a specific actor or actress from a multimedia datathrough recognition. Further, the processor 120 may, even when aspecific actor or actress is starring in both movies and dramas,identify the multimedia data as one of the movies based on userinformation stored in the storage.

Further, the processor 120 may, even when no content is identified,control the communicator comprising the circuitry 110 to transmit theuser information stored in the storage to the third server, and acquirean advertisement corresponding to the user information from the thirdserver via the communicator comprising the circuitry 110. For example,when a large number of viewing histories of movies are present in theuser viewing history, the third server may provide a movie trailer, newmovie information, etc., to the electronic apparatus 100 regardless of atype of content viewed by a current viewer.

In an embodiment, the third server may be a server separate from thefirst server and the second server. However, embodiments are not limitedthereto, and the third server may be the same server as at least one ofthe first server or the second server.

The electronic apparatus 100 may further include a storage and mayidentify an object in at least one frame based on an object recognitionmodel stored in the storage. In an embodiment, the object recognitionmodel may be acquired by training a plurality of sample images and aplurality of objects included in the plurality of sample images throughan artificial intelligence algorithm.

According to an embodiment, the storage may be an element separate fromthe processor 120. However, embodiments are not limited thereto, and astorage in which an object recognition model may be provided within theprocessor 120. Alternatively, the processor 120 itself may beimplemented in hardware to correspond to the object recognition model.Hereinafter, for the convenience of explanation, it will be assumed thatthe storage and the processor 120 are separate elements.

The object recognition model may be trained in another electronicapparatus and acquired. However, embodiments are not limited thereto,and the electronic apparatus 100 may directly acquire the objectrecognition model by training a plurality of sample images and aplurality of objects included in the plurality of sample images throughan artificial intelligence algorithm.

Further, the processor 120 may retrain the object recognition modelbased on information on the object and content. For example, theprocessor 120 may, when a monkey is repeatedly identified, acquire anartificial intelligence algorithm with improved identification speed andaccuracy of monkey by retraining.

The processor 120 may apply an optical character reader (OCR) to atleast one frame from among a plurality of frames and identify a text,and identify a content based on the identified text.

The processor 120 may compare the identified text with content guideinformation and acquire content information. Alternatively, theprocessor 120 may not compare the identified text with the content guideinformation and may identify the identified text itself as the contentinformation.

The electronic apparatus 100 may further include a display and maycontrol the display to sequentially display a plurality of frames, andidentify an object in a displayed frame from among the plurality offrames.

For example, the processor 120 may identify the object after a videoframe is acquired and displayed, rather than identifying the object asthe video frame is acquired. That is, the processor 120 may identify theobject only when the user views a content corresponding to the videoframe. A content viewing history of the user may be acquired throughthis operation.

The object described above may include at least one of a title ofcontent corresponding to at least one frame, a reproduction time of thecontent, channel information of the content, or a character included inat least one frame.

FIG. 1B is a block diagram illustrating in detail an exampleconfiguration of an electronic apparatus 100. The electronic apparatus100 may include a communicator circuitry 110, and a processor 120.Referring to FIG. 1B, the electronic apparatus 100 may include a storage130, a display 140, a user interface part 150, a speaker 160, a button170 and a microphone 180. Detailed descriptions of constitutionalelements illustrated in FIG. 1B that are redundant with constitutionalelements in FIG. 1A are omitted.

The processor 120 may control the overall operations of the electronicapparatus 100 using various programs stored in the storage 130.

In detail, the processor 120 may include a random access memory (RAM)121, a read only memory (ROM) 122, a main central processing unit (CPU)123, first through nth interfaces 124-1 through 124-n, and a bus 125.

The RAM 121, the ROM 122, the main CPU 123, the first through nthinterface 124-1 through 124-n, etc., may be connected to each other viathe bus 125.

The first to the nth interfaces 124-1 to 124-n may be connected to thevarious elements described above. One of the interfaces may be a networkinterface which is connected to an external apparatus via a network.

The main CPU 123 may access the storage 130, and perform booting usingan operating system (O/S) stored in the storage 130. In addition, themain CPU 123 may perform various operations using various programsstored in the storage 130.

The ROM 122 may store a set of instructions for system booting. When aturn-on command is input and power is supplied, the main CPU 123 may,according to an instruction stored in the ROM 122, copy the O/S storedin the storage 130 to the RAM 121, and execute O/S to boot the system.If the booting is completed, the main CPU 123 may copy variousapplication programs stored in the storage 130 to the RAM 121 andexecute the application programs copied to the RAM 121, therebyperforming various operations.

The main CPU 123 may provide a screen including various objects such asan icon, an image, text and the like. The main CPU 123 may acquire anattribute value such as a coordinate value at which each object will beindicated, form, size and color according to a screen layout based on anacquired control command. The main CPU 123 may provide a screen ofvarious layouts including an object based on the acquired attributevalue. The provided screen is displayed in a display area of the display140.

The processor 120 may perform processing on audio data. The processor120 may perform various processing, such as decoding, amplification, andnoise filtering of the audio data.

Further, the processor 120 may perform processing on multimedia data.The processor 120 may perform various kinds of image processing such asa decoding, a scaling, a noise filtering, a frame rate converting, aresolution converting, and the like, on multimedia data.

The operation of the above-described processor 120 may be performed by aprogram stored in the storage 130.

The storage 130 may store a variety of data, such as an operating system(O/S) software module for operating the electronic apparatus 100, anobject recognition module, an object recognition artificial intelligencemodule, an artificial intelligence training module or an opticalcharacter recognition (OCR) module.

The communicator comprising the circuitry 110 is an element to performcommunication with various types of external apparatuses according tovarious types of communication methods. The communicator comprising thecircuitry 110 may include a Wi-Fi chip 111, a Bluetooth chip 112, awireless communication chip 113 and a near field communication (NFC)chip 114.

The Wi-Fi chip 111 and the Bluetooth chip 112 may perform communicationaccording to a Wi-Fi method and a Bluetooth method, respectively. In acase in which the Wi-Fi chip 111 or the Bluetooth chip 112 is used, avariety of access information such as SSID, a session key, and the like,may be first transmitted and acquired, a communication access may beperformed using the variety of access information, and a variety ofinformation may be then transmitted and acquired. The wirelesscommunication chip 113 indicates a chip which performs communication inaccordance with various communication standards such as IEEE, Zigbee,3rd generation (3G), 3rd generation partnership project (3GPP), and longterm evolution (LTE) or the like. The NFC chip 114 means a chip which isoperated in the NFC scheme that uses a frequency band of 13.56 MHz amongvarious RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860to 960 MHz, 2.45 GHz, and the like.

Further, the communicator comprising the circuitry 110 may furtherinclude a wired communication interface such as HDMI, MHL, USB, DP,thunderbolt, RGB, D-SUB, DVI and the like. The processor 120 may beconnected to an external apparatus through a wired communicationinterface of the communicator comprising the circuitry 110. In thiscase, the processor 120 may acquire a multimedia data from the externalapparatus through the wired communication interface.

The display 140 may be implemented as various types of displays, such asa liquid crystal display (LCD), an organic light emitting diodes (OLED)display, and a plasma display panel (PDP). The display 140 may furtherinclude a driver circuit that may be realized as an amorphous-siliconthin film transistor (a-si TFT), low temperature poly silicon (LTPS)thin film transistor (TFT), or organic TFT (OTFT), and a backlight unit.The display 140 may be a touch screen including a touch sensor.

The user interface part 150 may acquire various user interactions. In anembodiment, the user interface part 150 may be implemented in variousforms according to implementing embodiments of the electronic apparatus100. For example, the user interface part 150 may be implemented as abutton provided on the electronic apparatus 100, a microphone acquiringa user speech, a camera detecting a user motion, etc. Further, when theelectronic apparatus 100 is implemented to be a mobile terminal based ontouch, the user interface part 150 may be implemented to be touch screenthat forms an interlayer structure with a touch pad. The user interfacepart 150 may be used as the above-described display 140.

The speaker 160 outputs various audio data processed by the processor120 and various notification sounds or voice messages, etc.

The button 170 may include various types of buttons, such as amechanical button, a touch pad, a wheel, etc., which are formed on thefront, side, or rear of the exterior of a main body of the electronicapparatus 100.

The microphone 180 acquires a user speech or other sounds and convertsthe user speech or other sounds into audio data.

FIG. 1C is a block diagram illustrating another example configuration ofan electronic apparatus. Referring to FIG. 1C, the electronic apparatus100 may include a communicator circuitry 110, and a processor 120, andthe processor 120 may include a storage 130, in which an objectrecognition model is stored. That is, the processor 120 may bemanufactured in the form of an on-chip including an object recognitionmodel. In an embodiment, the storage 130 may be implemented as cachememory, a register file and a buffer.

Through the method described above, the processor 120 may identify acontent with minimal fingerprint acquisition.

Hereinafter, an operation of the electronic apparatus 100 will bedescribed in greater detail with reference to the accompanying drawings.

FIG. 2 is a diagram briefly illustrating an entire system, according toan embodiment.

Referring to FIG. 2, the electronic apparatus 100 may be connected to adisplay apparatus 200, an external apparatus 300, a first server 400-1,a second server 400-2 and a third server 400-3.

The processor 120 may transmit multimedia data acquired from theexternal apparatus 300 to the display apparatus 200. The processor 120may identify an object from the multimedia data, and identify a contentbased on content guide information provided from the first server 400-1.In an embodiment, the content guide information may be informationstored in the electronic apparatus 100 before the multimedia data isacquired. However, embodiments are not limited thereto, and theprocessor 120 may, when the multimedia data is acquired, request contentguide information to the first server 400-1 and acquire the requestedcontent guide information.

Further, the processor 120 may, when no content is recognized and theexternal apparatus 300 is a predetermined apparatus, acquire afingerprint based on at least one frame included in the multimediacontent, transmit the acquired fingerprint to the second server 400-2,and acquire content information corresponding to the fingerprint fromthe second server 400-2.

Further, the processor 120 may transmit at least one of the identifiedobject or content to the third server 400-3, and acquire anadvertisement corresponding to at least one of the identified object orcontent from the third server 400-3. The processor 120 may transmit theacquired advertisement to the display apparatus 200.

According to an embodiment illustrated in FIG. 2, the electronicapparatus 100 and the display apparatus 200 are separate from eachother, but embodiments are not limited thereto. For example, theelectronic apparatus 100 and the display apparatus 200 may beimplemented in one apparatus. Alternatively, the electronic apparatus100 may be implemented as a USB and may be used as being connected tothe display apparatus 200.

Further, according to an embodiment illustrated in FIG. 2, the firstserver 400-1, the second server 400-2 and the third server 400-3 areseparate from each other. However, at least two of the first server400-1, the second server 400-2 and the third server 400-3 may beimplemented in one server.

FIG. 3A is a diagram illustrating an operation according to objectidentification, according to an embodiment. FIG. 3B is a diagramillustrating an operation according to object identification, accordingto an embodiment. FIG. 3C is a diagram illustrating an operationaccording to object identification, according to an embodiment.

The processor 120 may identify an object in at least one frame fromamong a plurality of frames included in the multimedia data. Forexample, the processor 120 may identify a monkey in a frame, asillustrated in FIG. 3A.

However, embodiments are not limited thereto, and the processor 120 mayidentify at least one of a title of content, a reproduction time ofcontent, channel information of content, a character included in atleast one frame, a logo, a content image, a type of external apparatus,a postal code, a keyword, a genre, a viewing rate or a review.

Meanwhile, the processor 120 may identify an object in a plurality offrames included in the multimedia content. For example, the processor120 may, when a monkey as in FIG. 3A is identified in a predeterminednumber of frames, identify a content corresponding to the monkey basedon content guide information. In an embodiment, the plurality of framesmay be consecutive frames.

Alternatively, the processor 120 may identify an object in a specificarea of a frame. For example, the processor 120 may divide the frameinto 3×3 areas and identify an object in the middle area.

In FIG. 3A, a screen in which additional information is not included ina content is illustrated, but embodiments are not limited thereto. Forexample, an external apparatus may transmit a multimedia data in whichon screen display (OSD) information is included in a frame as in FIG. 3Ato the electronic apparatus 100. In this case, the processor 120 mayapply optical character recognition (OCR) to the frame and acquire OSDinformation. Alternatively, the processor 120 may perform both OSDinformation recognition and object recognition. For example, theprocessor 120 may identify a text included in the frame as OSDinformation, and identify the monkey through object recognition.Further, the processor 120 may identify a content based on theidentified text or the identified monkey.

According to an embodiment, the processor 120 may identify an objectexcluding some areas in which OSD information is included. For example,when OSD information is included at a lower end of the frame, theprocessor 120 may identify the object in the remaining area other thanthe lower end of the frame in which the OSD information is stored.

The processor 120 may, when the object is identified, identify a contentcorresponding to the object based on content guide information. Forexample, the processor 120 may identify a global documentary (Hello!Monkey) corresponding to the monkey based on content guide informationas illustrated in FIG. 3B.

In FIG. 3B, a content guide information screen displayed to a viewer isillustrated and content guide information stored in the electronicapparatus 100 may be in a text format. Further, the content guideinformation may further include an image, and the processor 120 mayidentify an object from the image included in the content guideinformation, and compare an object identified in the frame with anobject identified in the image included in the content guide informationand identify a content.

The processor 120 may control the communicator comprising the circuitry110 to transmit identification information of the content to the thirdserver 400-3, and acquire an advertisement corresponding to theidentification information of the content from the third server 400-3via the communicator comprising the circuitry 110. For example, theprocessor 120 may, as illustrated in FIG. 3C, control the communicatorcomprising the circuitry 110 to transmit information on the globaldocumentary (Hello! Monkey) identified in FIG. 3B, and acquire a traveladvertisement of a high relevance to the global documentary (Hello!Monkey) from the third server 400-3 via the communicator comprising thecircuitry 110.

However, embodiments are not limited thereto, and the processor 120 maytransmit the identified object and viewing history of the user as wellas the identification information of the content to the third server400-3.

In the example described above, the acquired advertisement is a productadvertisement corresponding to at least one of the identified object orthe identified content, but is not limited thereto. For example, theacquired advertisement may be a content related to the identifiedcontent. For example, the acquired advertisement may be a preview of thenext episode of the identified content.

FIG. 4 is a flowchart illustrating a process of a content recognitionmethod, according to an embodiment.

The processor 120 may first identify an object in at least one framefrom among a plurality of frames included in the multimedia data, atoperation S410. Further, when the object is identified, S410-Y, theprocessor 120 may identify a content based on the recognized object, atoperation S420. The processor 120 may, when the content is identified,S420-Y, terminate an operation.

Alternatively, the processor 120 may, when the object is not identified,S410-N or the content is not identified, S420-N, identify whether a typeof external apparatus is a predetermined type, at operation S430. Whenthe type of external apparatus is a predetermined type, S430-Y, theprocessor 120 may acquire a fingerprint in at least one frame from amongthe plurality of frames, at operation S440.

The processor 120 may transmit the fingerprint to a server, at operationS450, and acquire identification information of a content correspondingto the fingerprint from the server, at operation S460.

The processor 120 may, when the identification information of thecontent is acquired from the server, terminate an operation.

Alternatively, the processor 120 may, when the type of externalapparatus is not a predetermined type, S430-N, terminate the operation.This is a case where it is identified that even if the fingerprint istransmitted to the server, the identification information of the contentcannot be received. For example, the type of external apparatus may be asmartphone, a game console, or the like.

Meanwhile, the processor 120 may, when the object is not identified,S410-N, or the content is not identified, S420-N, skip the operationS430 of identifying whether the type of external apparatus is apredetermined type, and immediately acquire a fingerprint in at leastone frame from among the plurality of frames, at operation S440.

FIG. 5 is a block diagram illustrating a configuration of anotherelectronic apparatus 500, according to an embodiment. In an embodiment,the another electronic apparatus 500 may be an apparatus acquiring anobject recognition model through an artificial intelligence algorithm.

Referring to FIG. 5, the another electronic apparatus 500 may include atleast one of a learning part 510 or a response part 520.

The learning part 510 may provide or train an artificial intelligencemodel for identifying an object using learning data. The learning part510 may provide an identification model including identificationcriteria by using collected learning data.

The response part 520 may acquire an object included in a predeterminedimage using a predetermined data as an input data of the trainedartificial intelligence model.

According to an embodiment, the learning part 510 and the response part520 may be included in another external server 500. However, this isonly an example, and the learning part 510 and the response part 520 maybe mounted within the electronic apparatus 100. For example, at least apart of the learning part 510 and at least a part of the response part520 may be implemented as a software module or manufactured as at leastone hardware chip and mounted in the electronic apparatus 100. Forexample, at least one of the learning part 510 or the response part 520may be manufactured in the form of a hardware chip exclusive forartificial intelligence (AI) or may be manufactured as a part of aprevious general purpose processor (e.g., CPU or application processor)or a dedicated graphics processor (e.g., GPU) and mounted in the variouselectronic apparatuses described above. The hardware chip exclusive forartificial intelligence is an exclusive processor specialized forprobability calculation, which may show high parallel processingperformance as compared with a general purpose processor so thatcalculation operations in the artificial intelligence field such asmachine learning may be processed quickly. When the learning part 510and the response part 520 are implemented as a software module (or aprogram module including an instruction), the software module may bestored on non-transitory computer readable media. In this case, thesoftware module may be provided by an operating system (OS) or by apredetermined application. Alternatively, a part of the software modulemay be provided by the operating system (OS) and the remaining part maybe provided by the predetermined application.

In this case, the learning part 510 and the response part 520 may bemounted in one electronic apparatus or may be respectively mounted inadditional electronic apparatuses. For example, one of the learning part510 or the response part 520 may be included in the electronic apparatus100 and the remaining one may be included in the another electronicapparatus 500. The learning part 510 and the response part 520 mayprovide the model information constructed by the learning part 510 tothe response part 520 via wire or wirelessly, or the data input to thelearning part 520 may be provided to the learning part 510 as additionallearning data.

FIG. 6 is a block diagram illustrating the learning part 510, accordingto an embodiment.

Referring to FIG. 6, the learning part 510 according to one or moreembodiments may include a learning data acquisition part 510-1 and amodel learning part 510-4. In addition, the learning part 510 mayfurther selectively include at least one of a learning datapreprocessing part 510-2, a learning data selection part 510-3, or amodel evaluation part 510-5.

The learning data acquisition part 510-1 may acquire a learning datanecessary for an artificial intelligence model for identifying anobject. In an embodiment, the learning data acquisition part 510-1 mayacquire a plurality of sample images and a plurality of objects includedin the plurality of sample images as learning data. The learning datamay be a data collected or tested by the learning part or themanufacturer of the learning part 510.

The model learning part 510-4 may train an artificial intelligence modelto include criteria for identifying an object from an image, using thelearning data. For example, the model learning part 510-4 may train anartificial intelligence model through supervised learning using at leasta part of the learning data. Alternatively, the model learning part510-4 may, for example, train itself using a learning data withoutspecial supervision so that an artificial intelligence model may betrained through unsupervised learning discovering criteria foridentifying an object. Further, the model learning part 510-4 may, forexample, train an artificial intelligence model through reinforcementlearning which uses a feedback as to whether a result of responseprovision according to learning is correct. In addition, the modellearning part 510-4 may, for example, train an artificial intelligencemodel by using a learning algorithm including error back-propagation orgradient descent.

In addition, the model learning part 510-4 may learn criteria ofselection as to what learning data is to be used to identify an objectusing an input data.

The model learning part 510-4 may, when a plurality of pre-constructedartificial intelligence models are present, identify an artificialintelligence model with high relevancy between input learning data andbasic learning data as a data recognition model to train. In this case,the basic learning data may be pre-classified according to the type ofdata, and the artificial intelligence model may be pre-constructedaccording to the type of data.

When the artificial intelligence model is trained, the model learningpart 510-4 may store the trained artificial intelligence model. In thiscase, the model learning part 510-4 may store the trained artificialintelligence model in a memory of the another electronic apparatus 500.Alternatively, the model learning part 510-4 may store the trainedartificial intelligence model in a server connected to the anotherelectronic apparatus 500 via a wired or wireless network or in a memoryof an electronic apparatus.

The data learning part 510 may further include a learning datapreprocessing part 510-2 and a learning data selection part 510-3 toimprove a response result of an artificial intelligence model or to savetime or resources necessary for generating an artificial intelligencemodel.

The learning data preprocessing part 510-2 may preprocess the acquireddata so that the acquired data is utilized in learning to identify anobject from an image. That is, the learning data preprocessing part510-2 may process the acquired data to a predetermined format. Forexample, the learning data preprocessing part 510-2 may divide a sampleimage into a plurality of areas.

The learning data selection part 510-3 may select data necessary forlearning from among the data acquired by the learning data acquisitionpart 510-1 and the data preprocessed by the learning data preprocessingpart 510-2. The selected learning data may be provided to the modellearning part 510-4. The learning data selection part 510-3 may selectlearning data necessary for learning from among the acquired orprocessed data according to predetermined selection criteria. Inaddition, the learning data selection part 510-3 may select learningdata according to predetermined selection criteria by learning of themodel learning part 510-4.

The learning part 510 may further include a model evaluation part 510-5to improve a response result of the artificial intelligence model.

The model evaluation part 510-5 may input evaluation data to theartificial intelligence model, and if the response result output fromthe evaluation data does not satisfy predetermined criteria, allow themodel learning part 510-4 to train again. In this case, the evaluationdata may be a predefined data to evaluate the artificial intelligencemodel.

On the other hand, when there are a plurality of trained artificialintelligence models, the model evaluation part 510-5 may evaluatewhether each of the trained artificial intelligence models satisfies thepredetermined criteria and determine the model which satisfies thepredetermined criteria as the final artificial intelligence model. Inthis case, when there are a plurality of models satisfying thepredetermined criteria, the model evaluation part 510-5 may determineany one or a predetermined number of models previously set in descendingorder of the evaluation score as the final artificial intelligencemodel.

FIG. 7 is a block diagram illustrating the response part 520, accordingto an embodiment.

Referring to FIG. 7, the response part 520 according to one or moreembodiments may include an input data acquisition part 520-1 and aresponse result providing part 520-4.

Further, the response part 520 may further selectively include at leastone of an input data preprocessing part 520-2, an input data selectionpart 520-3, or a model update part 520-5.

The input data acquisition part 520-1 may acquire a data necessary foridentifying an object. The response result providing part 520-4 mayapply an input data acquired by the input data acquisition part 520-1 tothe trained artificial intelligence model as an input value, andidentify an object from an image. The response result providing part520-4 may apply a data selected by the input data preprocessing part520-2 or by the input data selection part 520-3 which will be describedlater, to the artificial intelligence model as an input value, andacquire the response result. The response result may be determined bythe artificial intelligence model.

According to an embodiment, the response result providing part 520-4 mayapply an artificial intelligence model identifying the object acquiredby the input data acquisition part 520-1, and identify an object from animage.

The response part 520 may further include an input data preprocessingpart 520-2 and an input data selection part 520-3 to improve a responseresult of an artificial intelligence model or to save time or resourcesnecessary for providing the response result.

The input data preprocessing part 520-2 may preprocess data acquired toidentify an object so that the acquired data may be used. That is, theinput data preprocessing part 520-2 may process the acquired data to apredefined format.

The input data selection part 520-3 may select a data necessary forproviding a response from among a data acquired by the input dataacquisition part 520-1 or a data preprocessed by the input datapreprocessing part 520-2. The selected data may be provided to theresponse result providing part 520-4. The input data selection part520-3 may select some or all of the acquired or preprocessed dataaccording to predetermined selection criteria for providing a response.In addition, the input data selection part 520-3 may select a dataaccording to predetermined selection criteria by training of the modellearning part 510-4.

The model update part 520-5 may control an artificial intelligence modelto be updated based on an evaluation of a response result provided bythe response result providing part 520-4. For example, the model updatepart 520-5 may provide a response result provided by the response resultproviding part 520-4 to the model learning part 510-4, and therebyrequest the model learning part 510-4 may further train or update theartificial intelligence model.

FIG. 8 is a diagram illustrating an example in which an electronicapparatus 100 and an external server S are interlocked with each otherto learn and identify data, according to an embodiment.

Referring to FIG. 8, the external server S may learn criteria foridentifying an object from an image, and the electronic apparatus 100may identify an object from an image based on a learning result of theserver S.

In this case, a model learning part 510-4 of the server S may perform afunction of the learning part 510 illustrated in FIG. 6. That is, themodel learning part 510-4 of the server S may learn criteria as to whichimage information is to be used to identify an object and as to how theobject is to be identified using the information described above.

Further, the response result providing part 520-4 of the electronicapparatus 100 may apply a data selected by the input data selection part520-3 to an artificial intelligence model provided by the server S, andidentify the object from the image. Alternatively, the response resultproviding part 520-4 of the electronic apparatus 100 may acquire theartificial intelligence model provided by the server S from the serverS, and identify the object from the image.

FIG. 9 is a flowchart illustrating a control method of an electronicapparatus, according to an embodiment.

First, a multimedia data may be acquired from an external apparatus, atoperation S910. Further, an object may be identified in at least oneframe from among a plurality of frames included in the multimedia data,at operation S920. Further, a content corresponding to the identifiedobject may be identified based on content guide information providedfrom a first server, at operation S930.

According to an embodiment, when no content is identified, an operationof acquiring a fingerprint based on at least one frame, an operation oftransmitting the acquired fingerprint to a second server and anoperation of acquiring content information corresponding to thefingerprint from the second server may be further included.

Further, the operation of acquiring the fingerprint may include, when nocontent is identified, identifying a type of external apparatus, andwhen the type of external apparatus is a predetermined type, acquiring afingerprint based on at least one frame.

Meanwhile, the operation of transmitting at least one of the identifiedobject or the identified content to the third server 400-3, and theoperation of acquiring an advertisement corresponding to at least one ofthe identified object or the identified content from the third server400-3 may be further included.

Further, the operation of identifying the object, S920, may includeidentifying the object in at least one frame based on an objectrecognition model, and the object recognition model may be acquired bytraining a plurality of sample images and a plurality of objectsincluded in the plurality of sample images through an artificialintelligence algorithm.

In an embodiment, an operation of retraining the object recognitionmodel based on information on the object and the content.

Meanwhile, the operation of identifying the object, S920, may includeapplying an optical character reader (OCR) to at least one frame fromamong a plurality of frames and identify a text. The operation ofidentifying the content, S930, may include identifying the content basedon the identified text.

Further, an operation of identifying a type of external apparatus may befurther included. The operation of identifying the object, S920, mayinclude, when a type of external apparatus is a predetermined type,identifying an object in at least one frame.

Meanwhile, an operation of sequentially displaying a plurality of framesmay be further included. The operation of identifying the object, S920,may include identifying an object in a displayed frame from among theplurality of frames.

Further, the object may include at least one of a title of contentcorresponding to at least one frame, a reproduction time of the content,channel information of the content, or a character included in at leastone frame.

In accordance with one or more embodiments, an electronic apparatus mayidentify a content based on an object included in at least one framefrom among a plurality of frames included in a multimedia data, therebyminimizing the use of an external server in content recognition.

Meanwhile, the one or more embodiments described above may beimplemented as a S/W program including one or more instructions storedon machine-readable (e.g., computer-readable) storage media. The machinemay be an apparatus which is capable of calling a stored instructionfrom the storage medium and operating according to the calledinstruction, and may include an electronic apparatus (e.g., anelectronic apparatus 100) according to the above-described embodiments.When the one or more instructions are executed by a processor, theprocessor may perform a function corresponding to the one or moreinstructions directly or using other components under the control of theprocessor. The one or more instructions may include a code which isgenerated or executed by a compiler or an interpreter. Themachine-readable storage media may be provided as non-transitory storagemedia. Herein, the term “non-transitory” only denotes that a storagemedium does not include a signal but is tangible, which does notdistinguish a case where data is semi-permanently stored in a storagemedium from a case where data is temporarily stored in a storage medium.

According to an embodiment, the method according to the one or moreembodiments described above may be provided as being included in acomputer program product. The computer program product may be tradedbetween a seller and a consumer as a product. The computer programproduct may be distributed online in the form of machine-readablestorage media (e.g., compact disc read only memory (CD-ROM)) or throughan application store (e.g., Play Store™). As for online distribution, atleast a part of the computer program product may be at least temporarilystored in a server of a manufacturer, a server of an application store,or a storage medium such as memory, or may be temporarily generated.

The one or more embodiments described above may be embodied in arecording medium that may be read by a computer or a similar device tothe computer by using software, hardware, or a combination thereof. Insome cases, embodiments described herein may be implemented by processoritself. In a software configuration, one or more embodiments describedin the specification such as a procedure and a function may be embodiedas separate software modules. Each of the software modules may performone or more functions and operations described in the specification.

Meanwhile, computer instructions for carrying out processing operationsof machine according to the one or more embodiments described above maybe stored in non-transitory computer-readable media. Computerinstructions stored on such non-transitory computer-readable media may,when executed by a processor or of a specific device, cause the specificdevice to perform processing operations in the machine according to thevarious example embodiments described above. The non-transitory computerreadable medium is not limited to a medium that permanently stores datatherein, e.g., a register, a cache, a memory, or the like, but can be amedium that semi-permanently stores data therein and is readable by adevice. For example, the non-transitory computer readable medium mayinclude a compact disc (CD), a digital versatile disc (DVD), a harddisc, a Blu-ray disc, a memory card, or a read only memory (ROM).

The respective components (e.g., module or program) according to the oneor more embodiments may include a single entity or a plurality ofentities, and some of the corresponding sub components described abovemay be omitted, or another sub component may be further added to the oneor more embodiments. Alternatively or additionally, some elements (forexample, modules or programs) may be integrated into one entity, and afunction performed by the respective elements before integration may beperformed in the same or similar manner. The module, a program, oroperations executed by other elements according to one or moreembodiments may be executed consecutively, in parallel, repeatedly, orheuristically, or at least some operations may be executed according toa different order, may be omitted, or the other operation may be addedthereto.

The foregoing embodiments and advantages are merely examples and are notto be construed as limiting the disclosure. The present teaching may bereadily applied to other types of devices. Also, the description of theone or more embodiments is intended to be illustrative, and not to limitthe scope of the claims, and many alternatives, modifications, andvariations will be apparent to those skilled in the art.

What is claimed is:
 1. An electronic apparatus, comprising: acommunicator comprising circuitry; and a processor configured to: obtainmultimedia data from an external apparatus via the communicator;identify an object in at least one frame from among a plurality offrames included in the multimedia data; compare the identified objectwith an electronic program guide (EPG) provided from a first server; andidentify a content corresponding to the identified object based oncomparing the identified object with the EPG provided from the firstserver without acquiring a fingerprint corresponding to the at least oneframe.
 2. The electronic apparatus as claimed in claim 1, wherein theprocessor is further configured to: control the communicator to transmiteither one or both of the identified object and the identified contentto a third server; and obtain an advertisement corresponding to eitherone or both of the identified object and the identified content from thethird server via the communicator.
 3. The electronic apparatus asclaimed in claim 1, further comprising: a storage, wherein the processoris further configured to: identify the object in the at least one framebased on an object recognition model stored in the storage, wherein theobject recognition model is obtained by training a plurality of sampleimages and a plurality of objects included in the plurality of sampleimages through an artificial intelligence algorithm.
 4. The electronicapparatus as claimed in claim 3, wherein the processor is furtherconfigured to retrain the object recognition model based on informationrelating to the object and the content.
 5. The electronic apparatus asclaimed in claim 1, wherein the processor is further configured to:apply an optical character reader (OCR) to the at least one frame, fromamong the plurality of frames, and identify a text; compare theidentified text with the EPG; and identify the content based oncomparing the identified text with the EPG.
 6. The electronic apparatusas claimed in claim 1, wherein the processor is further configured to:identify a type of the external apparatus; and based on the type of theexternal apparatus being a predetermined type, identify the object inthe at least one frame.
 7. The electronic apparatus as claimed in claim1, further comprising: a display, wherein the processor is furtherconfigured to: control the display to sequentially display the pluralityof frames; and identify the object in a displayed frame from among theplurality of frames.
 8. The electronic apparatus as claimed in claim 1,wherein the object comprises any one or any combination of a title of acontent corresponding to the at least one frame, a reproduction time ofthe content, channel information of the content, and a characterincluded in the at least one frame.
 9. A control method of an electronicapparatus, the control method comprising: obtaining multimedia data froman external apparatus; identifying an object in at least one frame fromamong a plurality of frames included in the multimedia data; comparingthe identified object with an electronic program guide (EPG) providedfrom a first server; and identifying a content corresponding to theidentified object based on comparing the identified object with the EPGprovided from the first server without acquiring a fingerprintcorresponding to the at least one frame.
 10. The control method asclaimed in claim 9, further comprising: transmitting either one or bothof the identified object or the identified content to a third server;and obtaining an advertisement corresponding to either one or both ofthe identified object or the identified content from the third server.11. The control method as claimed in claim 9, wherein the identifyingthe object comprises: identifying the object in the at least one framebased on an object recognition model, and wherein the object recognitionmodel is obtained by training a plurality of sample images and aplurality of objects included in the plurality of sample images throughan artificial intelligence algorithm.
 12. The control method as claimedin claim 11, further comprising: retraining the object recognition modelbased on information relating to the object and the content.
 13. Thecontrol method as claimed in claim 9, wherein the identifying the objectcomprises: applying an optical character reader (OCR) to the at leastone frame from among the plurality of frames, and identifying a text,comparing the identified text with the EPG, and identifying the contentbased on comparing the identified text with the EPG.
 14. The controlmethod as claimed in claim 9, further comprising: identifying a type ofthe external apparatus, wherein the identifying the object comprises,based on the type of the external apparatus being a predetermined type,identifying the object in the at least one frame.
 15. The control methodas claimed in claim 9, further comprising: sequentially displaying theplurality of frames, wherein the identifying the object comprisesidentifying the object in a displayed frame, from among the plurality offrames.
 16. The control method as claimed in claim 9, wherein the objectcomprises any one or any combination of a title of a contentcorresponding to the at least one frame, a reproduction time of thecontent, channel information of the content, and a character included inthe at least one frame.
 17. The electronic apparatus as claimed in claim1, wherein the identified object is one of an image depicting an itemdisplayed when the at least one frame is displayed or textualinformation displayed when the at least one frame is displayed.
 18. Theelectronic apparatus as claimed in claim 1, wherein the identifiedobject is an image depicting an item displayed when the at least oneframe is displayed.
 19. The electronic apparatus as claimed in claim 1,wherein the identified object is textual information displayed when theat least one frame is displayed.